Method for processing image through neural network and electronic device thereof

ABSTRACT

An electronic device may execute a neural network model for generating an image. Generating the image includes consecutively obtaining a plurality of frames by using a camera. A first frame among the plurality of frames may be presented to a first group of the neural network model, as a first input, and first result data corresponding to the first input stored in a memory. The first frame may be presented to a second group, as a second input. Upon determining that a second computation parameter of the second group is the same as a first computation parameter of the first group, the first result data is used as second result data corresponding to the second input, without performing a neural network computation that is based on the second group.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of InternationalApplication No. PCT/KR2022/009347 designating the United States, filedon Jun. 29, 2022, in the Korean Intellectual Property Receiving Officeand claiming priority to Korean Patent Application No. 10-2021-0084955,filed on Jun. 29, 2021, in the Korean Intellectual Property Office, thedisclosures of which are incorporated by reference herein in theirentireties.

TECHNICAL FIELD

Embodiments disclosed in the present document relate to a method ofprocessing an image through a neural network and an electronic devicethereof.

BACKGROUND ART

An artificial intelligent system (or an integrated intelligence system)is a computer system that implements human-level intelligence, and is asystem in which a machine learns and judges on its own. The more it isused, the more the accuracy, such as a recognition rate, improves. Anartificial intelligence technology includes a machine learning (e.g.,deep learning) technology that uses an algorithm that classify/learn thecharacteristics of input data by itself, and simulate a function such ascognition, judgment, etc. of a human brain by using a machine learningalgorithm.

The machine learning technology can be used for applications thatinclude, for example, at least one of a linguistic understandingtechnology of recognizing human language/script, a visual understandingtechnology of recognizing a thing like human eyes, areasoning/prediction technology of logically reasoning and predicting byjudging information, a knowledge expression technology of processinghuman experience information as knowledge data, and a motion controltechnology of controlling the autonomous driving of a vehicle and themovement of a robot.

An electronic device equipped with an artificial intelligent system cansupport a function of, in image processing, analyzing image dataobtained through a camera, and performing the image processing through aneural network, and performing one or more operations based on theanalyzed result.

DISCLOSURE Technical Problem

When performing an image processing that includes several procedures fora consecutively inputted multi-frame by using a neural network, anelectronic device can perform the same computation procedure in eachprocedure. However, when performing such an image processing for amulti-frame, the electronic device can perform a large amount ofcomputations even though it is not necessary to perform the samecomputation procedure(s).

Various embodiments of the present disclosure present a method ofanalyzing an image processing structure through a neural network,identifying a computation procedure being repeated and storing theresult of such a computation procedure, and using the stored result datawithout having to repeat the procedure, and an electronic devicethereof.

Technical problems to be achieved in the present disclosure are notlimited to the technical problems mentioned above, and other technicalproblems not mentioned can be clearly understood by those having anordinary skill in the art to which the present invention belongs fromthe description below.

Technical Solution

An electronic device of an embodiment disclosed in the present documentmay include a camera, a memory, and at least one processor. The at leastone processor may execute a neural network model for providing an image,the neural network model including a first group including at least afirst computation layer among a plurality of computation layers and asecond group including at least a second computation layer among theplurality of computation layers. The processor further consecutivelyobtains a plurality of frames by using the camera, and presents a firstframe among the plurality of frames to the first group of the neuralnetwork model, as a first input. The processor stores first result datacorresponding to the first input in the memory. The processor furtherpresents the first frame to the second group, as a second input, anddetermines whether a second computation parameter of the second group isthe same as a first computation parameter of the first group. If thesecond computation parameter and the first computation parameter are thesame, the processor obtain the first result data as second result datacorresponding to the second input, without performing a neural networkcomputation that is based on the second group. If the second computationparameter and the first computation parameter are different from eachother, the processor performs the neural network computation, based onthe computation parameter of the second group, and obtain second resultdata distinguished from the first result data.

A method of operating an electronic device of an embodiment disclosed inthe present document may include consecutively obtaining a plurality offrames by using a camera, and presenting a first frame among theplurality of frames to a first group of a neural network model, as afirst input, and storing first result data corresponding to the firstinput in a memory. The method further includes presenting the firstframe to a second group, as a second input, and determining whether asecond computation parameter of the second group is the same as a firstcomputation parameter of the first group. In response to the secondcomputation parameter and the first computation parameter being thesame, obtaining the first result data as second result datacorresponding to the second input, without performing a neural networkcomputation that is based on the second group, and, in response todetermining that the second computation parameter and the firstcomputation parameter are different from each other, the neural networkcomputation is performed based on the computation parameter of thesecond group to obtain second result data distinguished from the firstresult data.

Advantageous Effects

According to various embodiments disclosed in the present document,because a part of a previously executed computation is stored andreused, an amount of computation of a neural network can be reduced.

Also, according to various embodiments, because all computations of theneural network are not performed, processing speed is improved.

In addition, various effects directly or indirectly identified throughthe present document may be presented.

DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a structure of an electronic device anda camera according to an embodiment.

FIG. 2 illustrates a hardware construction and a software module of anelectronic device according to an embodiment.

FIG. 3 is a diagram schematically illustrating a procedure of executionof a neural network model when obtaining a plurality of image frames inan electronic device according to an embodiment.

FIG. 4 illustrates that image processing is performed for a plurality ofimage frames when a computation procedure of each group of a neuralnetwork model is the same in an electronic device according to anembodiment.

FIG. 5 is a flowchart for illustrating an operation procedure of aneural network model for providing a preview image in an electronicdevice according to an embodiment.

FIG. 6 is a flowchart illustrating a procedure of storing a computationresult through a neural network model in an electronic device accordingto an embodiment.

FIG. 7 is a flowchart illustrating a procedure of reusing a computationresult stored through a neural network model in an electronic deviceaccording to an embodiment.

FIG. 8 illustrates the processing of a case where groups of a neuralnetwork model include the same computation procedure in an electronicdevice according to an embodiment.

FIG. 9 illustrates the processing of a case where groups of a neuralnetwork model include the same computation procedure in an electronicdevice according to an embodiment.

FIG. 10 is a block diagram of an electronic device in a networkenvironment according to various embodiments.

FIG. 11 is a block diagram illustrating a camera module according tovarious embodiments.

MODE FOR INVENTION

Various embodiments of the present document will be described below withreference to the accompanying drawings. However, this is not intended tolimit the present document to specific embodiments, and should beunderstood to include various modifications, equivalents, and/oralternatives of an embodiment of the present invention.

FIG. 1 is a diagram illustrating a structure of an electronic device anda camera according to an embodiment.

FIG. 1 is a schematic diagram showing an external view of an electronicdevice 100 (e.g., an electronic device 1001 of FIG. 10 ) equipped with acamera 180 (e.g., a camera module 1080 of FIG. 10 ), and the camera 180according to an embodiment. Although the embodiment of FIG. 1 has beenillustrated and described as a mobile device, in particular, a smartphone, technical solutions described herein may be applied to anyelectronic device equipped with a camera.

Referring to FIG. 1 , a display 110 may be disposed on a front surfaceof the electronic device 100 of an embodiment. In an embodiment, thedisplay 110 may occupy most of the front surface of the electronicdevice 100. A display 110 and bezel 190 region surrounding at least someedges of the display 110 may be disposed on the front surface of theelectronic device 100. The display 110 may include a flat area, and acurved area extending from the flat area toward a side surface of theelectronic device 100. The electronic device 100 illustrated in FIG. 1is one example, and various embodiments are possible. For example, thedisplay 110 of the electronic device 100 may include only the flat areawithout the curved area or may include the curved area only at an edgeof one side instead of both sides. Also, in an embodiment, the curvedarea extends to a rear surface of the electronic device, so theelectronic device 100 may include an additional flat area.

In an embodiment, the electronic device 100 may additionally include aspeaker, a receiver, a front camera, a proximity sensor, a home key, andthe like. The electronic device 100 of an embodiment may be presented inwhich a rear cover 150 is integrated with a main body of the electronicdevice as well. In another embodiment, the rear cover 150 may beseparated from the main body of the electronic device 100, and have aform in which a battery may be replaced. The rear cover 150 may bereferred to as a battery cover or a rear cover as well.

In an embodiment, a fingerprint sensor 171 for recognizing a user'sfingerprint may be included in a first area 170 of the display 110.Because the fingerprint sensor 171 is disposed in a lower layer of thedisplay 110, the fingerprint sensor 171 may not be recognized by a useror may be difficult to be recognized. Also, in addition to thefingerprint sensor 171, a sensor for additional user/biometricauthentication may be disposed in a partial area of the display 110. Inanother embodiment, a sensor for user/biometric authentication may bedisposed in one area of a bezel 190. For example, an IR sensor for irisauthentication may be exposed through one area of the display 110 or maybe exposed through one area of the bezel 190.

In an embodiment, a front camera 161 may be disposed in a second area160 on the front surface of the electronic device 100. In the embodimentof FIG. 1 , the front camera 161 is illustrated as being exposed throughone area of the display 110, but in another embodiment, the front camera161 may be exposed through the bezel 190. The electronic device 100 mayinclude one or more front cameras 161. For example, the electronicdevice 100 may include two front cameras, such as a first front cameraand a second front camera. In an embodiment, the first front camera andthe second front camera may be cameras of the same type having the samespecification (e.g., a pixel), but the first front camera and the secondfront camera may be implemented as cameras of different specifications.The electronic device 100 may support a function (e.g., 3D imaging, autofocus, etc.) related to a dual camera through two front cameras. Theabove-mentioned description of the front camera may be equally orsimilarly applied to a rear camera of the electronic device 100.

In an embodiment, in the electronic device 100, various hardware orsensors 163 to assist photographing, such as a flash, may beadditionally disposed. For example, a distance sensor (e.g., a TOFsensor) for detecting a distance between a subject and the electronicdevice 100 may be further included. The distance sensor may be appliedto both a front camera and/or a rear camera. The distance sensor may beseparately disposed or included and be disposed in the front cameraand/or the rear camera.

In an embodiment, at least one physical key may be disposed on a sideportion of the electronic device 100. For example, a first function key151 for turning on/off the display 110 or turning on/off the power ofthe electronic device 100 may be disposed at a right edge with respectto the front surface of the electronic device 100. In an embodiment, asecond function key 152 for controlling a volume or a screen brightness,etc. of the electronic device 100 may be disposed at a left edge withrespect to the front surface of the electronic device 100. In additionto this, an additional button or key may be disposed even on the frontsurface or rear surface of the electronic device 100. For example, aphysical button or a touch button mapped to a specific function may bedisposed in a lower area of the front bezel 190.

The electronic device 100 illustrated in FIG. 1 corresponds to oneexample, and does not limit a shape of a device to which a technicalspirit disclosed in the present disclosure is applied. For example, byadopting a flexible display and a hinge structure, the technical spiritof the present disclosure may be applied even to a foldable electronicdevice that is foldable in a horizontal direction or is foldable in avertical direction, a rollable electronic device that is rollable, atablet or a notebook computer. In addition, the present technical spiritmay be applied even when it is possible that the first camera and thesecond camera facing the same direction are disposed to face differentdirections through rotation of the device, folding, deformation, etc.

Referring to FIG. 1 , the electronic device 100 (e.g., an electronicdevice 1001 of FIG. 10 ) of an embodiment may include the camera 180(e.g., a camera module 1080 of FIG. 10 ). The camera 180 may include alens assembly 111 (e.g., a lens assembly 1110 of FIG. 11 ), a housing113, an infrared cut filter 115, and an image sensor 120 (e.g., an imagesensor 1130 of FIG. 11 ) and an image signal processor 130 (e.g., animage signal processor 1160 of FIG. 11 ).

In an embodiment, the lens assembly 111 may have different number,arrangement, type, etc. of lenses according to the front camera and therear camera. According to the type of the lens assembly, the frontcamera and the rear camera may have different characteristics (e.g., afocal length, a maximum magnification, etc.). The lens may be movedforward and backward along an optical axis, and may operate wherein atarget object, which is a subject, may be clearly captured by changing afocal length.

In an embodiment, the camera 180 may include a housing 113 which mountsa barrel mounting at least one or more lenses aligned on the opticalaxis and at least one coil surrounding a periphery of the barrel withrespect to the optical axis.

In an embodiment, the infrared cut filter 115 may be disposed on anupper surface of the image sensor 120. An image of a subject passingthrough the lens may be partially filtered by the infrared cut filter115 and then be detected by the image sensor 120.

In an embodiment, the image sensor 120 may be disposed on an uppersurface of a printed circuit board. The image sensor 120 may beelectrically connected to the image signal processor 130 connected tothe printed circuit board 140 by a connector. A flexible printed circuitboard (FPCB) or a cable, etc. may be used as the connector.

In an embodiment, the image sensor 120 may be a complementary metaloxide semiconductor (CMOS) sensor or a charged coupled device (CCD)sensor. A plurality of individual pixels are integrated in the imagesensor 120, and each individual pixel may include a micro lens, a colorfilter, and a photodiode. Each individual pixel, which is a kind ofphotodetector, may convert inputted light into an electrical signal. Thephotodetector generally may not detect a wavelength of captured light byitself and may not determine color information. The photodetector mayinclude a photodiode.

In an embodiment, light information of a subject incident through thelens assembly 111 may be converted into an electrical signal by theimage sensor 120 and be input to the image signal processor 130.

In an embodiment, the camera 180 may be disposed on the front surface aswell as the rear surface of the electronic device 100. Also, theelectronic device 100 may include a plurality of cameras 180 as well asone camera 180 so as to improve camera performance. For example, theelectronic device 100 may further include the front camera 161 for videocall or self-camera photography. The front camera 161 may support arelatively small number of pixels compared to a rear camera module. Thefront camera may be relatively smaller than the rear camera module.

FIG. 2 illustrates a hardware construction and a software module of anelectronic device according to an embodiment. In a description of FIG. 2, the construction illustrated in FIG. 1 may be briefly described or adescription thereof may be omitted.

Referring to FIG. 2 , the electronic device 100 may use a hardwareand/or software module in order to support functions for applying aneffect to an image. For example, by executing instructions stored in amemory 230, the processor 210 may drive a neural network analysis module211 and a neural network execution module 213. In various embodiments,software modules other than those shown in FIG. 2 may be implemented.For example, at least two modules may be integrated into one module, orone module may be divided into two or more modules. In addition, workperformance may be improved by sharing a function between hardware andsoftware modules. For example, the electronic device 100 may includeboth an encoder implemented as hardware and an encoder implemented as asoftware module, and a part of data obtained through at least one cameramodule may be processed in the hardware encoder, and the other part maybe processed in the software encoder. Also, a connection relationshipbetween the hardware/software illustrated in FIG. 2 is for descriptionconvenience's sake, and does not limit a flow/direction of data orcommands. Components included in the electronic device 100 may havevarious electrical/operative connection relationships.

In an embodiment, the processor 210 may obtain an image frame (or aframe) through the camera 180. The processor 210 may consecutively (orsequentially) obtain a plurality of frames (or multi-frames) through thecamera 180. For example, the processor 210 may sequentially obtain annth frame, an n+1st frame, and an n+2nd frame.

In an embodiment, the processor 210 may provide a result image by usingthe plurality of obtained frames. In some embodiments, a subset of theobtained frames may be used to generate the result image. In someembodiments, more than one result image is generated using differentcombinations of the obtained frames. For example, the processor 210 mayprovide a first result image by using the nth frame to an n+4th frame.The processor 210 may provide a second result image by using the n+1stframe to an n+5th frame.

In an embodiment, the processor 210 may use a neural network model togenerate the result image(s). According to an embodiment, the neuralnetwork model may include at least one group. At least one group of anembodiment may each include at least one layer. The at least one layermay be a linear layer (e.g., a convolution layer) or a non-linear layer(e.g., a polling layer). In an embodiment, the processor 210 may performa neural network computation through a convolution layer. In anembodiment, the convolution layer may provide a feature vector, based onan inputted value. For example, the convolution layer may provide afeature vector corresponding to an inputted image signal.

According to an embodiment, a neural network may be included in anexternal device capable of communicating with the electronic device 100.For example, the neural network may be included in an intelligent serverincluding the neural network and/or a structure for neural networklearning. According to an embodiment, the processor 210 may transmitand/or receive data to and from an intelligent server that uses a neuralnetwork by using a communication circuit included in the electronicdevice 100. In an embodiment, the intelligent server may include astructure for using an artificial intelligent (AI) system. Theartificial intelligent system may be a neural network-based system(e.g., a feedforward neural network (FNN), and/or a recurrent neuralnetwork (RNN)).

According to an embodiment, the processor 210 may perform a functionrelated to artificial intelligence by using the intelligent server. Forexample, the processor 210 may perform an image providing operation,alone or together with the intelligent server, based on the obtainedimage signal.

In an embodiment, the processor 210 may perform image processing for anobtained image frame through a computation structure of groups (orlayers) of the neural network model. For example, the processor 210 maysequentially perform computation processing for an nth frame through afirst group 211_1, a second group 211_2, a third group 211_3, a fourthgroup 211_4, and/or a fifth group 211_5 of the neural network model.

In an embodiment, the memory 230 may store programs and/or data. Forexample, the memory 230 may store a computation parameter for a neuralnetwork, input data, and output data. In an embodiment, the processor210 may perform a computation operation (e.g., a convolutioncomputation), based on input data received from the memory 230, andstore the computation result in the memory 230.

In an embodiment, the neural network analysis module 211 may analyze astructure of a neural network model. When the neural network modelperforms image processing for multiple frames including at least two ormore frames, the neural network analysis module 211 may analyze adependency between neural network groups processing each frame, and/orthe match or non-match of a computation procedure. The neural networkanalysis module 211 may forward analyzed information to the neuralnetwork execution module 213.

In an embodiment, by using the dependency between the neural networkgroups and/or association information forwarded from the neural networkanalysis module 211, the neural network execution module 213 may store acomputation procedure and/or a computation result, and reuse the storedprocedure and/or result.

In an embodiment, the processor 210 may display the provided resultimage through at least a portion of the display 110. The processor 210may output a plurality of generated result images as preview images. Theprocessor 210 may display a continuous image such as a moving picture,based on the plurality of generated result images.

FIG. 3 is a diagram schematically illustrating a procedure of executionof a neural network model when an electronic device obtains a pluralityof image frames according to an embodiment.

In an embodiment, the processor 210 may input an nth frame 301 to afirst group 310 at time t. The processor 210 may perform a process forthe nth frame 301 through a computation procedure of the first group310. The processor 210 may input the nth frame 301 to a second group 320and input an n+1st frame 303 to the first group 310, at time t+1.

In an embodiment, the processor 210 may store, in the memory 230, n_1stdata 231 including a computation result and/or computation procedure ofthe first group 310 for the nth frame 301. This may be identicallyapplied even to an n+1st frame. For example, the processor 210 maystore, in the memory 230, n+1_1st data 232 including a computationresult and/or computation procedure of the first group 310 for the n+1stframe 303. According to an embodiment, since the computation structuresor computation procedures of the first group 310 and the second group320 are the same, the processor 210 may use the same (already computed)result for the nth frame 301 in the second group 320, in a step in whichthe nth frame 301 must go through the second group 320. That thecomputation structures or computation procedures are the same may beunderstood as, for example, that a convolution computation process isthe same and a weight value and a bias value used for convolutioncomputation are the same. In another embodiment, when the computationstructures of the first group 310 and the second group 320 are not thesame, the processor 210 may input, to the second group 320, the nthframe 301 determined in the first group 310, without using the storedcomputation result. The above description may be equally applied even tothe n+1st frame 303.

Below, the weight value and the bias value may be expressed as a weightvalue and a bias value, respectively.

FIG. 4 illustrates image processing for a plurality of image frames whena computation procedure of each group of a neural network model is thesame in an electronic device according to the embodiment.

Referring to FIG. 4 , in an embodiment, computation structures and/orcomputation procedures of the first group 410 to the fourth group 440may be the same. Convolution weight value and bias value used in thecomputation procedures (e.g., convolution computation) of the firstgroup 410 to the fourth group 440 may be the same.

In an embodiment, when consecutively inputted frames (e.g., an nthframe, an n+1st frame, an n+2nd frame, and an n+3rd frame) aresequentially inputted from the first group, the processor 210 mayconsecutively store the computation result of the first group 410, inthe memory (e.g., a DRAM) 230. For example, when the nth frame isinputted to the first group 410 at time t, the processor 210 may storethe computation result of the nth frame of the first group 410, in thememory 230. When the nth frame is inputted to the second group 420 andthe n+1st frame is inputted to the first group 410 at time t+1, theprocessor 210 may store the computation result of the n+1st frame of thefirst group 410, in the memory 230.

In an embodiment, when frames inputted to the first group 410 aresequentially performed in the second group 420, the third group 430, andthe fourth group 440, the processor 210 may obtain and use thecomputation result stored in the memory 230, without performing actualcomputation. For example, when the nth frame undergoes the computationof the fourth group 440, the computation result of the first group 410for the nth frame may be obtained, and the computation result may beused as the computation result of the fourth group 440.

In an embodiment, after the frames go through up to the fourth group,one result image frame may be generated by the fifth group. In otherwords, one result image frame may be generated using the plurality ofimage frames (e.g., the nth frame to the n+4th frame) obtained throughthe camera 180. In an embodiments, different number of groups may exist.

FIG. 5 is a flowchart illustrating an operation procedure of a neuralnetwork model for providing a preview image in an electronic deviceaccording to an embodiment.

In operation 510 of an embodiment, the processor 210 may execute aneural network model for providing a preview image. The neural networkmodel may be a complex network model formed by a plurality of layersconnected to each other. In an embodiment, the processor 210 may trainthe neural network model, and execute the obtained neural network modelafter the training is finished, and use for image processing. Accordingto an embodiment, the neural network model may include at least onegroup and may include a structure for obtaining a plurality of inputdata. For example, the neural network model may include a first groupand a second group. Hereinafter, it has been described that the neuralnetwork model includes the first group and the second group fordescription convenience's sake, but according to various embodiments,the number of groups included in the neural network model is not limitedthereto. According to an embodiment, each group included in the neuralnetwork model may receive data and use a computation parameter (e.g.,weight value, bias value), and may output result data through acomputation procedure that is based on each group. For example, thefirst group may obtain a first input and output first result data. Also,the second group may obtain a second input and output second resultdata.

In operation 520 of an embodiment, the processor 210 may consecutivelyobtain a plurality of frames by using the camera 180. The processor 210may consecutively obtain the plurality of frames according to a framerate of an image sensor.

In operation 530 of an embodiment, the processor 210 may present a firstframe among the plurality of frames to the first group of the neuralnetwork model, as a first input.

In operation 540 of an embodiment, the processor 210 may store the firstresult data corresponding to the first input in the memory 230. Theprocessor 210 may output the first result data based on the first framethat is processed through the first group of the neural network model.The processor 210 may store the outputted first result data in thememory (e.g., a DRAM).

In operation 550 of an embodiment, the processor 210 may determinewhether a second computation parameter of the second group and a firstcomputation parameter of the first group are the same (e.g., equal). Forexample, the processor 210 may determine whether weight values and biasvalues used in the computation (e.g., convolution computation) of theneural network model are the same as each other.

In operation 560 of an embodiment, the processor 210 may obtain and usethe first result data stored in the memory 230, as the second resultdata, without performing a neural network computation that is based onthe second group.

In operation 570 of an embodiment, the processor 210 may present thefirst frame to the second group, as a second input. According to anembodiment, in operation 580, the processor 210 may perform a neuralnetwork computation, based on the computation parameter of the secondgroup, and obtain the second result datadistinguished/different/distinct from the first result data.

In other words, when the computation parameter of the first group andthe computation parameter of the second group are the same/match/areequal, the processor 210 may obtain the first result data (computedusing the first frame through the first group) as the second result dataof the second group. As another example, when the computation parameterof the first group and the computation parameter of the second group aredifferent from each other, the processor 210 presents the first frame tothe second group, as the second input, and obtains the second resultdata in which the neural network computation is performed based on thecomputation parameter of the second group.

FIG. 6 is a flowchart illustrating a procedure of storing a computationresult through a neural network model in an electronic device accordingto an embodiment.

In operation 610 of an embodiment, the processor 210 may analyze acomputation structure of the neural network model. The processor 210 mayanalyze a computation structure of each group included in the neuralnetwork model.

In operation 620 of an embodiment, the processor 210 may determinewhether each computation group of the neural network model isindependent and the same computation procedure exists. For example, theprocessor 210 may determine whether the first group and the second groupof the neural network model are independent computation groups. Thesecond group may be a computation group in which specific data isprocessed after the first group. For example, the processor 210 maydetermine whether the computation procedures of the first group and thesecond group of the neural network model are the same as each other. Forexample, the processor 210 may determine whether weight values and biasvalues used in the computation (e.g., convolution computation)procedures of the neural network model are the same as each other.

In operation 630 of an embodiment, the processor 210 may store thecomputation procedure. The processor 210 may provide reuse informationabout the computation procedure. When the computation procedures of thefirst group and the second group of the neural network model are thesame as each other, and the weight values and the bias values used inthe computation procedures are the same as each other, the processor 210may provide reuse information about the computation procedure and/orcomputation result of at least one group (e.g., the first group or thesecond group).

FIG. 7 is a flowchart illustrating a procedure of reusing a computationresult stored through a neural network model in an electronic deviceaccording to an embodiment.

In operation 710 of an embodiment, the processor 210 may determinewhether a computation result is reusable. The processor 210 may analyzea computation structure of a neural network and determine whether thecomputation result is reusable. When the computation result is notreusable, the processor 210 may perform operation 760, and when thecomputation result is reusable, the processor 210 may perform operation720.

In operation 720 of an embodiment, the processor 210 may determinewhether the computation result has been stored in the memory 230. Forexample, the processor 210 may determine whether the reusablecomputation result has been stored in the memory 230. According to anembodiment, when the reusable computation result has been stored in thememory 230, the processor 210 may perform operation 750, and when thecomputation result has not been stored in the memory 230, the processor210 may perform operation 730 and operation 740.

According to an embodiment, when the reusable computation result has notbeen stored in the memory 230, in operation 730, the processor 210 mayperform a general neural network execution procedure. Here, the generalneural network procedure may mean performing image processing through aneural network, without using the computation result stored in thememory 230.

According to an embodiment, in operation 740, the processor 210 mayperform a general neural network procedure and store the determinationresult in the memory 230.

In operation 750 of an embodiment, the processor 210 may obtain thestored computation result from the memory 230. The processor 210 mayperform the following computation, based on the computation resultobtained from the memory. For example, the processor 210 may output astored result of the first group of the neural network as a result ofthe second group.

FIG. 8 illustrates the processing of a case where groups of a neuralnetwork model include the same computation procedure in an electronicdevice according to an embodiment.

Referring to FIG. 8 , the neural network model may include a first group810, a second group 820, a third group 830, a fourth group 840, and afifth group 850. The groups 810 to 850 may include a plurality oflayers. In FIG. 8 , each layer may mean a computation procedure and/or acomputation parameter.

Referring to FIG. 8 , in an embodiment, the first group 810 and thesecond group 820 of the neural network model may include the samecomputation procedure. For example, layers included in the first group810 and the second group 820 of the neural network model may be the sameas each other. In an embodiment, the second group 820 and the thirdgroup 830 of the neural network model may partially include the samecomputation procedure. For example, the layers of the second group 820may be the same as at least some of layers of the third group 830. In anembodiment, the third group 830 and the fourth group 840 of the neuralnetwork model may include the same computation procedure. For example,layers included in the third group 830 and the fourth group 840 of theneural network model may be the same as each other.

In an embodiment, when image processing is performed by sequentiallypassing an input frame through each group 810 to 850, the processor 210may use the determination result of a previous group, when computationprocedures of each group are the same as each other, and weight valuesand bias values used in the computation procedures are the same as eachother.

In an embodiment, referring to FIG. 8 , when an nth frame is used as aninput of the second group 820, the processor 210 may use the determinedresult and/or computation procedure of the first group 810 as thedetermined result and/or computation procedure of the second group 820.For example, when the first group 810 and the second group 820 includethe same computation procedure, and weight values and bias values usedin the computation procedure (e.g., convolution computation) are thesame as each other, the processor 210 may use the determined resultand/or computation procedure of the first group 810 as the result and/orcomputation procedure of the second group 820, without performing acomputation on the second group 820 when the nth frame is used as theinput of the second group 820.

In an embodiment, referring to FIG. 8 , when the nth frame is used as aninput of the third group 830, the processor 210 may partially use thedetermined result and/or computation procedure of the first group 810 inthe third group 830. For example, when the second group 820 and thethird group 830 include the partially same computation procedure, andweight values and bias values used in the computation procedure (e.g.,convolution computation) are the same as each other, the processor 210may use the determined result and/or computation procedure of the firstgroup 810 as the intermediate determination result and/or intermediatecomputation procedure of the second group 820 when the nth frame is usedas the input of the third group 830.

In an embodiment, referring to FIG. 8 , when the nth frame is used as aninput of the fourth group 820, the processor 210 may use the determinedresult and/or computation procedure of the third group 810 as thedetermined result and/or computation procedure of the fourth group 840.For example, when the third group 830 and the fourth group 840 includethe same computation procedure, and weight values and bias values usedin the computation procedure (e.g., convolution computation) are thesame as each other, the processor 210 may use the determined resultand/or computation procedure of the third group 830 as the determinedresult and/or computation procedure of the fourth group 840, withoutperforming a computation of the fourth group having the nth frame as aninput when the nth frame is used as the input of the fourth group 840.

Although the description of FIG. 8 above uses the nth frame, it may beidentically applied to an n+1st frame, an n+2nd frame, and an n+3rdframe consecutively obtained after the nth frame.

FIG. 9 illustrates a case where groups of a neural network model includethe same computation procedure in an electronic device according to anembodiment.

Referring to FIG. 9 , the neural network model may include a first group910, a second group 920, a third group 930, a fourth group 940, and afifth group 950. The groups 910 to 950 may include a plurality oflayers. In FIG. 9 , each layer may mean a computation procedure and/or acomputation parameter.

Referring to FIG. 9 , in an embodiment, computation procedures of thefirst group 910 to the fourth group 940 of the neural network model maybe different from each other. In other words, layers included in thefirst group 910 to the fourth group 940 may not be the same as eachother. In an embodiment, the first group 910 and the third group 930 ofthe neural network model may be partially the same as each other. Forexample, layers of the first group 910 may be the same as at least someof layers of the third group 930. In an embodiment, the second group 920and the fourth group 940 of the neural network model may be partiallythe same as each other. For example, layers of the second group 920 maybe the same as at least some of layers of the fourth group 940.

In an embodiment, when image processing is performed by sequentiallypassing an input frame through each group 910 to 950, the processor 210may use the determined result of a previous group, when computationprocedures of each group are the same as each other, and weight valuesand bias values used in the computation procedures are the same as eachother.

In an embodiment, referring to FIG. 9 , the processor 210 may computeand process an nth frame as an input of the first group 910. Since thecomputation procedures of the first group 910 and the second group 920are not the same or the weight values and bias values used in thecomputation procedures are not the same, the processor 210 may not usethe determined result and/or computation procedure of the first group910, in the second group 920. In other words, the processor 210 maycompute and process the computation result of the first group 910 of thenth frame as an input of the second group 920.

In an embodiment, referring to FIG. 9 , when the nth frame is used as aninput of the third group 930, the processor 210 may use the determinedresult and/or computation procedure of the first group 910 as theintermediate computation result and/or intermediate computationprocedure of the third group 930. For example, when computationprocedures of the first group 910 and the third group 930, and/or weightvalues and bias values used in the computation procedures (e.g.,convolution computation) are matched with each other only in a specificportion, the processor 210 may reuse the determined result and/orcomputation procedure of the first group 910 up to the matching portion,without performing the same portion of the third group 930 as that ofthe first group 910.

In an embodiment, referring to FIG. 9 , when the nth frame is used as aninput of the fourth group 940, the processor 210 may use the determinedresult and/or computation procedure of the second group 920 as theintermediate computation result and/or intermediate computationprocedure of the fourth group 940.

Although the description of FIG. 9 above uses the nth frame, it may beidentically applied to the n+1st frame, the n+2nd frame, and the n+3rdframe consecutively obtained after the nth frame.

FIG. 10 is a block diagram illustrating an electronic device 1001 in anetwork environment 1000 according to various embodiments.

Referring to FIG. 10 , the electronic device 1001 in the networkenvironment 1000 may communicate with an electronic device 1002 via afirst network 1098 (e.g., a short-range wireless communication network),or at least one of an electronic device 1004 or a server 1008 via asecond network 1099 (e.g., a long-range wireless communication network).According to an embodiment, the electronic device 1001 may communicatewith the electronic device 1004 via the server 1008. According to anembodiment, the electronic device 1001 may include a processor 1020,memory 1030, an input module 1050, a sound output module 1055, a displaymodule 1060, an audio module 1070, a sensor module 1076, an interface1077, a connecting terminal 1078, a haptic module 1079, a camera module1080, a power management module 1088, a battery 1089, a communicationmodule 1090, a subscriber identification module (SIM) 1096, or anantenna module 1097. In some embodiments, at least one of the components(e.g., the connecting terminal 1078) may be omitted from the electronicdevice 1001, or one or more other components may be added in theelectronic device 1001. In some embodiments, some of the components(e.g., the sensor module 1076, the camera module 1080, or the antennamodule 1097) may be implemented as a single component (e.g., the displaymodule 1060).

The processor 1020 may execute, for example, software (e.g., a program1040) to control at least one other component (e.g., a hardware orsoftware component) of the electronic device 1001 coupled with theprocessor 1020, and may perform various data processing or computation.According to one embodiment, as at least part of the data processing orcomputation, the processor 1020 may store a command or data receivedfrom another component (e.g., the sensor module 1076 or thecommunication module 1090) in volatile memory 1032, process the commandor the data stored in the volatile memory 1032, and store resulting datain non-volatile memory 1034. According to an embodiment, the processor1020 may include a main processor 1021 (e.g., a central processing unit(CPU) or an application processor (AP)), or an auxiliary processor 1023(e.g., a graphics processing unit (GPU), a neural processing unit (NPU),an image signal processor (ISP), a sensor hub processor, or acommunication processor (CP)) that is operable independently from, or inconjunction with, the main processor 1021. For example, when theelectronic device 1001 includes the main processor 1021 and theauxiliary processor 1023, the auxiliary processor 1023 may be adapted toconsume less power than the main processor 1021, or to be specific to aspecified function. The auxiliary processor 1023 may be implemented asseparate from, or as part of the main processor 1021.

The auxiliary processor 1023 may control at least some of functions orstates related to at least one component (e.g., the display module 1060,the sensor module 1076, or the communication module 1090) among thecomponents of the electronic device 1001, instead of the main processor1021 while the main processor 1021 is in an inactive (e.g., sleep)state, or together with the main processor 1021 while the main processor1021 is in an active state (e.g., executing an application). Accordingto an embodiment, the auxiliary processor 1023 (e.g., an image signalprocessor or a communication processor) may be implemented as part ofanother component (e.g., the camera module 1080 or the communicationmodule 1090) functionally related to the auxiliary processor 1023.According to an embodiment, the auxiliary processor 1023 (e.g., theneural processing unit) may include a hardware structure specified forartificial intelligence model processing. An artificial intelligencemodel may be generated by machine learning. Such learning may beperformed, e.g., by the electronic device 1001 where the artificialintelligence is performed or via a separate server (e.g., the server1008). Learning algorithms may include, but are not limited to, e.g.,supervised learning, unsupervised learning, semi-supervised learning, orreinforcement learning. The artificial intelligence model may include aplurality of artificial neural network layers. The artificial neuralnetwork may be a deep neural network (DNN), a convolutional neuralnetwork (CNN), a recurrent neural network (RNN), a restricted boltzmannmachine (RBM), a deep belief network (DBN), a bidirectional recurrentdeep neural network (BRDNN), deep Q-network or a combination of two ormore thereof but is not limited thereto. The artificial intelligencemodel may, additionally or alternatively, include a software structureother than the hardware structure.

The memory 1030 may store various data used by at least one component(e.g., the processor 1020 or the sensor module 1076) of the electronicdevice 1001. The various data may include, for example, software (e.g.,the program 1040) and input data or output data for a command relatedthereto. The memory 1030 may include the volatile memory 1032 or thenon-volatile memory 1034.

The program 1040 may be stored in the memory 1030 as software, and mayinclude, for example, an operating system (OS) 1042, middleware 1044, oran application 1046.

The input module 1050 may receive a command or data to be used byanother component (e.g., the processor 1020) of the electronic device1001, from the outside (e.g., a user) of the electronic device 1001. Theinput module 1050 may include, for example, a microphone, a mouse, akeyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The sound output module 1055 may output sound signals to the outside ofthe electronic device 1001. The sound output module 1055 may include,for example, a speaker or a receiver. The speaker may be used forgeneral purposes, such as playing multimedia or playing record. Thereceiver may be used for receiving incoming calls. According to anembodiment, the receiver may be implemented as separate from, or as partof the speaker.

The display module 1060 may visually provide information to the outside(e.g., a user) of the electronic device 1001. The display module 1060may include, for example, a display, a hologram device, or a projectorand control circuitry to control a corresponding one of the display,hologram device, and projector. According to an embodiment, the displaymodule 1060 may include a touch sensor adapted to detect a touch, or apressure sensor adapted to measure the intensity of force incurred bythe touch.

The audio module 1070 may convert a sound into an electrical signal andvice versa. According to an embodiment, the audio module 1070 may obtainthe sound via the input module 1050, or output the sound via the soundoutput module 1055 or a headphone of an external electronic device(e.g., an electronic device 1002) directly (e.g., wiredly) or wirelesslycoupled with the electronic device 1001.

The sensor module 1076 may detect an operational state (e.g., power ortemperature) of the electronic device 1001 or an environmental state(e.g., a state of a user) external to the electronic device 1001, andthen generate an electrical signal or data value corresponding to thedetected state. According to an embodiment, the sensor module 1076 mayinclude, for example, a gesture sensor, a gyro sensor, an atmosphericpressure sensor, a magnetic sensor, an acceleration sensor, a gripsensor, a proximity sensor, a color sensor, an infrared (IR) sensor, abiometric sensor, a temperature sensor, a humidity sensor, or anilluminance sensor.

The interface 1077 may support one or more specified protocols to beused for the electronic device 1001 to be coupled with the externalelectronic device (e.g., the electronic device 1002) directly (e.g.,wiredly) or wirelessly. According to an embodiment, the interface 1077may include, for example, a high definition multimedia interface (HDMI),a universal serial bus (USB) interface, a secure digital (SD) cardinterface, or an audio interface.

A connecting terminal 1078 may include a connector via which theelectronic device 1001 may be physically connected with the externalelectronic device (e.g., the electronic device 1002). According to anembodiment, the connecting terminal 1078 may include, for example, aHDMI connector, a USB connector, a SD card connector, or an audioconnector (e.g., a headphone connector).

The haptic module 1079 may convert an electrical signal into amechanical stimulus (e.g., a vibration or a movement) or electricalstimulus which may be recognized by a user via his tactile sensation orkinesthetic sensation. According to an embodiment, the haptic module1079 may include, for example, a motor, a piezoelectric element, or anelectric stimulator.

The camera module 1080 may capture a still image or moving images.According to an embodiment, the camera module 1080 may include one ormore lenses, image sensors, image signal processors, or flashes.

The power management module 1088 may manage power supplied to theelectronic device 1001. According to one embodiment, the powermanagement module 1088 may be implemented as at least part of, forexample, a power management integrated circuit (PMIC).

The battery 1089 may supply power to at least one component of theelectronic device 1001. According to an embodiment, the battery 1089 mayinclude, for example, a primary cell which is not rechargeable, asecondary cell which is rechargeable, or a fuel cell.

The communication module 1090 may support establishing a direct (e.g.,wired) communication channel or a wireless communication channel betweenthe electronic device 1001 and the external electronic device (e.g., theelectronic device 1002, the electronic device 1004, or the server 1008)and performing communication via the established communication channel.The communication module 1090 may include one or more communicationprocessors that are operable independently from the processor 1020(e.g., the application processor (AP)) and supports a direct (e.g.,wired) communication or a wireless communication. According to anembodiment, the communication module 1090 may include a wirelesscommunication module 1092 (e.g., a cellular communication module, ashort-range wireless communication module, or a global navigationsatellite system (GNSS) communication module) or a wired communicationmodule 1094 (e.g., a local area network (LAN) communication module or apower line communication (PLC) module). A corresponding one of thesecommunication modules may communicate with the external electronicdevice via the first network 1098 (e.g., a short-range communicationnetwork, such as Bluetooth™, wireless-fidelity (Wi-Fi) direct, orinfrared data association (IrDA)) or the second network 1099 (e.g., along-range communication network, such as a legacy cellular network, a5G network, a next-generation communication network, the Internet, or acomputer network (e.g., LAN or wide area network (WAN)). These varioustypes of communication modules may be implemented as a single component(e.g., a single chip), or may be implemented as multi components (e.g.,multi chips) separate from each other. The wireless communication module1092 may identify and authenticate the electronic device 1001 in acommunication network, such as the first network 1098 or the secondnetwork 1099, using subscriber information (e.g., international mobilesubscriber identity (IMSI)) stored in the subscriber identificationmodule 1096.

The wireless communication module 1092 may support a 5G network, after a4G network, and next-generation communication technology, e.g., newradio (NR) access technology. The NR access technology may supportenhanced mobile broadband (eMBB), massive machine type communications(mMTC), or ultra-reliable and low-latency communications (URLLC). Thewireless communication module 1092 may support a high-frequency band(e.g., the mmWave band) to achieve, e.g., a high data transmission rate.The wireless communication module 1092 may support various technologiesfor securing performance on a high-frequency band, such as, e.g.,beamforming, massive multiple-input and multiple-output (massive MIMO),full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, orlarge scale antenna. The wireless communication module 1092 may supportvarious requirements specified in the electronic device 1001, anexternal electronic device (e.g., the electronic device 1004), or anetwork system (e.g., the second network 1099). According to anembodiment, the wireless communication module 1092 may support a peakdata rate (e.g., 20 Gbps or more) for implementing eMBB, loss coverage(e.g., 164 dB or less) for implementing mMTC, or U-plane latency (e.g.,0.5 ms or less for each of downlink (DL) and uplink (UL), or a roundtrip of 1 ms or less) for implementing URLLC.

The antenna module 1097 may transmit or receive a signal or power to orfrom the outside (e.g., the external electronic device) of theelectronic device 1001. According to an embodiment, the antenna module1097 may include an antenna including a radiating element composed of aconductive material or a conductive pattern formed in or on a substrate(e.g., a printed circuit board (PCB)). According to an embodiment, theantenna module 1097 may include a plurality of antennas (e.g., arrayantennas). In such a case, at least one antenna appropriate for acommunication scheme used in the communication network, such as thefirst network 1098 or the second network 1099, may be selected, forexample, by the communication module 1090 (e.g., the wirelesscommunication module 1092) from the plurality of antennas. The signal orthe power may then be transmitted or received between the communicationmodule 1090 and the external electronic device via the selected at leastone antenna. According to an embodiment, another component (e.g., aradio frequency integrated circuit (RFIC)) other than the radiatingelement may be additionally formed as part of the antenna module 1097.

According to various embodiments, the antenna module 1097 may form ammWave antenna module. According to an embodiment, the mmWave antennamodule may include a printed circuit board, a RFIC disposed on a firstsurface (e.g., the bottom surface) of the printed circuit board, oradjacent to the first surface and capable of supporting a designatedhigh-frequency band (e.g., the mmWave band), and a plurality of antennas(e.g., array antennas) disposed on a second surface (e.g., the top or aside surface) of the printed circuit board, or adjacent to the secondsurface and capable of transmitting or receiving signals of thedesignated high-frequency band.

At least some of the above-described components may be coupled mutuallyand communicate signals (e.g., commands or data) therebetween via aninter-peripheral communication scheme (e.g., a bus, general purposeinput and output (GPIO), serial peripheral interface (SPI), or mobileindustry processor interface (MIPI)).

According to an embodiment, commands or data may be transmitted orreceived between the electronic device 1001 and the external electronicdevice 1004 via the server 1008 coupled with the second network 1099.Each of the electronic devices 1002 or 1004 may be a device of a sametype as, or a different type, from the electronic device 1001. Accordingto an embodiment, all or some of operations to be executed at theelectronic device 1001 may be executed at one or more of the externalelectronic devices 1002, 1004, or 1008. For example, if the electronicdevice 1001 should perform a function or a service automatically, or inresponse to a request from a user or another device, the electronicdevice 1001, instead of, or in addition to, executing the function orthe service, may request the one or more external electronic devices toperform at least part of the function or the service. The one or moreexternal electronic devices receiving the request may perform the atleast part of the function or the service requested, or an additionalfunction or an additional service related to the request, and transferan outcome of the performing to the electronic device 1001. Theelectronic device 1001 may provide the outcome, with or without furtherprocessing of the outcome, as at least part of a reply to the request.To that end, a cloud computing, distributed computing, mobile edgecomputing (MEC), or client-server computing technology may be used, forexample. The electronic device 1001 may provide ultra low-latencyservices using, e.g., distributed computing or mobile edge computing. Inanother embodiment, the external electronic device 1004 may include aninternet-of-things (IoT) device. The server 1008 may be an intelligentserver using machine learning and/or a neural network. According to anembodiment, the external electronic device 1004 or the server 1008 maybe included in the second network 1099. The electronic device 1001 maybe applied to intelligent services (e.g., smart home, smart city, smartcar, or healthcare) based on 5G communication technology or IoT-relatedtechnology.

FIG. 11 is a block diagram 1100 illustrating the camera module 1080according to various embodiments. Referring to FIG. 11 , the cameramodule 1080 may include a lens assembly 1110, a flash 1120, an imagesensor 1130, an image stabilizer 1140, memory 1150 (e.g., buffermemory), or an image signal processor 1160. The lens assembly 1110 maycollect light emitted or reflected from an object whose image is to betaken. The lens assembly 1110 may include one or more lenses. Accordingto an embodiment, the camera module 1080 may include a plurality of lensassemblies 1110. In such a case, the camera module 1080 may form, forexample, a dual camera, a 360-degree camera, or a spherical camera. Someof the plurality of lens assemblies 1110 may have the same lensattribute (e.g., view angle, focal length, auto-focusing, f number, oroptical zoom), or at least one lens assembly may have one or more lensattributes different from those of another lens assembly. The lensassembly 1110 may include, for example, a wide-angle lens or a telephotolens.

The flash 1120 may emit light that is used to reinforce light reflectedfrom an object. According to an embodiment, the flash 1120 may includeone or more light emitting diodes (LEDs) (e.g., a red-green-blue (RGB)LED, a white LED, an infrared (IR) LED, or an ultraviolet (UV) LED) or axenon lamp. The image sensor 1130 may obtain an image corresponding toan object by converting light emitted or reflected from the object andtransmitted via the lens assembly 1110 into an electrical signal.According to an embodiment, the image sensor 1130 may include oneselected from image sensors having different attributes, such as a RGBsensor, a black-and-white (BW) sensor, an IR sensor, or a UV sensor, aplurality of image sensors having the same attribute, or a plurality ofimage sensors having different attributes. Each image sensor included inthe image sensor 1130 may be implemented using, for example, a chargedcoupled device (CCD) sensor or a complementary metal oxide semiconductor(CMOS) sensor.

The image stabilizer 1140 may move the image sensor 1130 or at least onelens included in the lens assembly 1110 in a particular direction, orcontrol an operational attribute (e.g., adjust the read-out timing) ofthe image sensor 1130 in response to the movement of the camera module1080 or the electronic device 1001 including the camera module 1080.This allows compensating for at least part of a negative effect (e.g.,image blurring) by the movement on an image being captured. According toan embodiment, the image stabilizer 1140 may sense such a movement bythe camera module 1080 or the electronic device 1001 using a gyro sensor(not shown) or an acceleration sensor (not shown) disposed inside oroutside the camera module 1080. According to an embodiment, the imagestabilizer 1140 may be implemented, for example, as an optical imagestabilizer.

The memory 1150 may store, at least temporarily, at least part of animage obtained via the image sensor 1130 for a subsequent imageprocessing task. For example, if image capturing is delayed due toshutter lag or multiple images are quickly captured, a raw imageobtained (e.g., a Bayer-patterned image, a high-resolution image) may bestored in the memory 1150, and its corresponding copy image (e.g., alow-resolution image) may be previewed via the display device 1060.Thereafter, if a specified condition is met (e.g., by a user's input orsystem command), at least part of the raw image stored in the memory1150 may be obtained and processed, for example, by the image signalprocessor 1160. According to an embodiment, the memory 1150 may beconfigured as at least part of the memory 1030 or as a separate memorythat is operated independently from the memory 1030.

The image signal processor 1160 may perform one or more image processingwith respect to an image obtained via the image sensor 1130 or an imagestored in the memory 1150. The one or more image processing may include,for example, depth map generation, three-dimensional (3D) modeling,panorama generation, feature point extraction, image synthesizing, orimage compensation (e.g., noise reduction, resolution adjustment,brightness adjustment, blurring, sharpening, or softening). Additionallyor alternatively, the image signal processor 1160 may perform control(e.g., exposure time control or read-out timing control) with respect toat least one (e.g., the image sensor 1130) of the components included inthe camera module 1080. An image processed by the image signal processor1160 may be stored back in the memory 1150 for further processing, ormay be provided to an external component (e.g., the memory 1030, thedisplay device 1060, the electronic device 1002, the electronic device1004, or the server 1008) outside the camera module 1080. According toan embodiment, the image signal processor 1160 may be configured as atleast part of the processor 1020, or as a separate processor that isoperated independently from the processor 1020. If the image signalprocessor 1160 is configured as a separate processor from the processor1020, at least one image processed by the image signal processor 1160may be displayed, by the processor 1020, via the display device 1060 asit is or after being further processed.

According to an embodiment, the electronic device 1001 may include aplurality of camera modules 1080 having different attributes orfunctions. In such a case, at least one of the plurality of cameramodules 1080 may form, for example, a wide-angle camera and at leastanother of the plurality of camera modules 1080 may form a telephotocamera. Similarly, at least one of the plurality of camera modules 1080may form, for example, a front camera and at least another of theplurality of camera modules1080 may form a rear camera.

As described above, an electronic device (e.g., the electronic device100 of FIG. 2 ) of an embodiment may include a camera (e.g., the camera180 of FIG. 2 ), a memory (e.g., the memory 230 of FIG. 2 ), and atleast one processor (e.g., the processor 210 of FIG. 2 ). The at leastone processor may execute a neural network model for providing an image,the neural network model including a first group (e.g., the first group211_1 of FIG. 2 ) including at least a first computation layer among aplurality of computation layers and a second group (e.g., the secondgroup 211_2 of FIG. 2 ) including at least a second computation layeramong the plurality of computation layers. The at least one processormay consecutively obtain a plurality of frames by using the camera. Theat least one processor may present a first frame among the plurality offrames to the first group of the neural network model, as a first input,and store first result data corresponding to the first input in thememory. The at least one processor may present the first frame to thesecond group, as a second input, and determine whether a secondcomputation parameter of the second group is the same as a firstcomputation parameter of the first group. The at least one processor, inresponse to the second computation parameter and the first computationparameter being the same, obtain (outputs, uses, accesses, provides,substitutes) the first result data as second result data correspondingto the second input, without performing a neural network computationthat is based on the second group. The at least one processor, inresponse to the second computation parameter and the first computationparameter being different from each other, performs the neural networkcomputation, based on the computation parameter of the second group, andobtains second result data. The second result data may be distinct(different) from the first result data.

According to an embodiment, the neural network model may further includea third group including at least a third computation layer among theplurality of computation layers. The at least one processor may presenta second frame consecutive to the first frame among the plurality offrames to the first group of the neural network model, as a third input,and obtain third result data corresponding to the third input. The atleast one processor may provide the image, as a result image, based onthe second result data of the first frame and the third result data ofthe second frame. The result image is generated based on the secondresult data and the third result data.

According to an embodiment, the at least one processor may store thethird result data in the memory.

According to an embodiment, the electronic device may further include adisplay, and the at least one processor may displays the result imagethrough the display.

According to an embodiment, the at least one processor may determinewhether a computation structure of the first group and a computationstructure of the second group are the same as each other.

According to an embodiment, in response to the computation structure ofthe first group and the computation structure of the second group beingthe same, the at least one processor may omit (bypass, skip) theexecution of a neural network computation that is based on the secondgroup.

According to an embodiment, the first computation parameter and thesecond computation parameter may be at least one of a weight valueand/or a bias value which are used for a convolution computation.

According to an embodiment, when a computation structure of the firstgroup is the same as at least a portion of a computation structure ofthe second group, the at least one processor may obtain the first resultdata as intermediate result data of up to the same portion as the atleast portion of the second group.

According to an embodiment, the first computation layer and the secondcomputation layer may include at least a convolution layer.

According to an embodiment, the at least one processor may provide reuseinformation about the first result data, in storing the first resultdata in the memory.

As described above, a method of operating an electronic device mayinclude consecutively obtaining a plurality of frames by using a camera,and presenting a first frame among the plurality of frames to a firstgroup of a neural network model, as a first input, and storing firstresult data corresponding to the first input in a memory. The methodfurther includes presenting the first frame to a second group, as asecond input, and determining whether a second computation parameter ofthe second group is the same as a first computation parameter of thefirst group. The method further includes, in response to determiningthat the second computation parameter and the first computationparameter are the same as each other, obtaining the first result data assecond result data corresponding to the second input, without performinga neural network computation that is based on the second group. Themethod further includes, in response to determining that the secondcomputation parameter and the first computation parameter are differentfrom each other, performing the neural network computation, based on thecomputation parameter of the second group, and obtaining second resultdata. The second result data may be distinguished (different) from thefirst result data.

According to an embodiment, the method may include presenting a secondframe consecutive to the first frame among the plurality of frames tothe first group of the neural network model, as a third input, andobtaining third result data corresponding to the third input. The methodfurther includes generating an image, based on the second result data ofthe first frame and the third result data of the second frame.

According to an embodiment, the method may include storing the thirdresult data in the memory.

According to an embodiment, the method may include displaying the imagethrough a display.

According to an embodiment, the method may include determining whether acomputation structure of the first group and a computation structure ofthe second group are the same as each other.

According to an embodiment, the method may include, in response todetermining that the computation structure of the first group and thecomputation structure of the second group are the same as each other,omitting the execution of a neural network computation that is based onthe second group.

According to an embodiment, the first computation parameter and thesecond computation parameter may be at least one of a weight value and abias value which are used for a convolution computation.

According to an embodiment, the method may include, in response todetermining that a computation structure of the first group is the sameas at least a portion of a computation structure of the second group,obtaining the first result data as intermediate result data of up to thesame portion as the at least portion of the second group.

According to an embodiment, a first computation layer included in thefirst group and a second computation layer included in the second groupmay include at least a convolution layer.

According to an embodiment, the method may include providing reuseinformation about the first result data, in storing the first resultdata in the memory.

The electronic device according to various embodiments may be one ofvarious types of electronic devices. The electronic devices may include,for example, a portable communication device (e.g., a smartphone), acomputer device, a portable multimedia device, a portable medicaldevice, a camera, a wearable device, or a home appliance. According toan embodiment of the disclosure, the electronic devices are not limitedto those described above.

It should be appreciated that various embodiments of the presentdisclosure and the terms used therein are not intended to limit thetechnological features set forth herein to particular embodiments andinclude various changes, equivalents, or replacements for acorresponding embodiment. With regard to the description of thedrawings, similar reference numerals may be used to refer to similar orrelated elements. It is to be understood that a singular form of a nouncorresponding to an item may include one or more of the things, unlessthe relevant context clearly indicates otherwise. As used herein, eachof such phrases as “A or B,” “at least one of A and B,” “at least one ofA or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least oneof A, B, or C,” may include any one of, or all possible combinations ofthe items enumerated together in a corresponding one of the phrases. Asused herein, such terms as “1st” and “2nd,” or “first” and “second” maybe used to simply distinguish a corresponding component from another,and does not limit the components in other aspect (e.g., importance ororder). It is to be understood that if an element (e.g., a firstelement) is referred to, with or without the term “operatively” or“communicatively”, as “coupled with,” “coupled to,” “connected with,” or“connected to” another element (e.g., a second element), it means thatthe element may be coupled with the other element directly (e.g.,wiredly), wirelessly, or via a third element.

As used in connection with various embodiments of the disclosure, theterm “module” may include a unit implemented in hardware, software, orfirmware, and may interchangeably be used with other terms, for example,“logic,” “logic block,” “part,” or “circuitry”. A module may be a singleintegral component, or a minimum unit or part thereof, adapted toperform one or more functions. For example, according to an embodiment,the module may be implemented in a form of an application-specificintegrated circuit (ASIC).

Various embodiments as set forth herein may be implemented as software(e.g., the program 1040) including one or more instructions that arestored in a storage medium (e.g., internal memory 1036 or externalmemory 1038) that is readable by a machine (e.g., the electronic device1001). For example, a processor (e.g., the processor 1020) of themachine (e.g., the electronic device 1001) may invoke at least one ofthe one or more instructions stored in the storage medium, and executeit, with or without using one or more other components under the controlof the processor. This allows the machine to be operated to perform atleast one function according to the at least one instruction invoked.The one or more instructions may include a code generated by a complieror a code executable by an interpreter. The machine-readable storagemedium may be provided in the form of a non-transitory storage medium.Wherein, the term “non-transitory” simply means that the storage mediumis a tangible device, and does not include a signal (e.g., anelectromagnetic wave), but this term does not differentiate betweenwhere data is semi-permanently stored in the storage medium and wherethe data is temporarily stored in the storage medium.

According to an embodiment, a method according to various embodiments ofthe disclosure may be included and provided in a computer programproduct. The computer program product may be traded as a product betweena seller and a buyer. The computer program product may be distributed inthe form of a machine-readable storage medium (e.g., compact disc readonly memory (CD-ROM)), or be distributed (e.g., downloaded or uploaded)online via an application store (e.g., PlayStore™), or between two userdevices (e.g., smart phones) directly. If distributed online, at leastpart of the computer program product may be temporarily generated or atleast temporarily stored in the machine-readable storage medium, such asmemory of the manufacturer's server, a server of the application store,or a relay server.

According to various embodiments, each component (e.g., a module or aprogram) of the above-described components may include a single entityor multiple entities, and some of the multiple entities may beseparately disposed in different components. According to variousembodiments, one or more of the above-described components may beomitted, or one or more other components may be added. Alternatively oradditionally, a plurality of components (e.g., modules or programs) maybe integrated into a single component. In such a case, according tovarious embodiments, the integrated component may still perform one ormore functions of each of the plurality of components in the same orsimilar manner as they are performed by a corresponding one of theplurality of components before the integration. According to variousembodiments, operations performed by the module, the program, or anothercomponent may be carried out sequentially, in parallel, repeatedly, orheuristically, or one or more of the operations may be executed in adifferent order or omitted, or one or more other operations may beadded.

1. An electronic device comprising: a camera; a memory; and at least oneprocessor, wherein the at least one processor is configured to: executea neural network model for providing an image, the neural network modelcomprising a first group comprising at least a first computation layeramong a plurality of computation layers and a second group comprising atleast a second computation layer among the plurality of computationlayers; consecutively obtain a plurality of frames by using the camera;present a first frame among the plurality of frames to the first groupof the neural network model, as a first input; store first result datacorresponding to the first input in the memory; present the first frameto the second group, as a second input; determine whether a secondcomputation parameter of the second group is the same as a firstcomputation parameter of the first group; in response to determiningthat the second computation parameter and the first computationparameter are the same, obtain the first result data as second resultdata corresponding to the second input, without performing a neuralnetwork computation that is based on the second group; in response todetermining that the second computation parameter and the firstcomputation parameter are different from each other, perform the neuralnetwork computation, based on the computation parameter of the secondgroup, and obtain second result data distinguished from the first resultdata.
 2. The electronic device of claim 1, wherein the neural networkmodel further comprises a third group comprising at least a thirdcomputation layer among the plurality of computation layers, and the atleast one processor: presents a second frame consecutive to the firstframe among the plurality of frames to the first group of the neuralnetwork model, as a third input; obtains third result data correspondingto the third input; and generates an image, based on the second resultdata of the first frame and the third result data of the second frame.3. The electronic device of claim 2, wherein the at least one processorstores the third result data in the memory.
 4. The electronic device ofclaim 2, further comprising a display, wherein the at least oneprocessor displays the image through the display.
 5. The electronicdevice of claim 1, wherein the at least one processor determines whethera computation structure of the first group and a computation structureof the second group are the same as each other.
 6. The electronic deviceof claim 5, wherein, in response to determining that the computationstructure of the first group and the computation structure of the secondgroup are the same as each other, the at least one processor omits theexecution of a neural network computation that is based on the secondgroup.
 7. The electronic device of claim 1, wherein the firstcomputation parameter and the second computation parameter are at leastone of a weight value and/or a bias value which are used for aconvolution computation.
 8. The electronic device of claim 1, wherein,in response to determining that a computation structure of the firstgroup is the same as at least a portion of a computation structure ofthe second group, the at least one processor obtains the first resultdata as intermediate result data of up to the same portion as the atleast portion of the second group.
 9. The electronic device of claim 1,wherein the first computation layer and the second computation layercomprise at least a convolution layer.
 10. The electronic device ofclaim 1, wherein the at least one processor provides reuse informationabout the first result data, in storing the first result data in thememory.
 11. A method of operating an electronic device, the methodcomprising: consecutively obtaining a plurality of frames by using acamera; presenting a first frame among the plurality of frames to afirst group of a neural network model, as a first input; storing firstresult data corresponding to the first input in a memory; presenting thefirst frame to a second group, as a second input; determining whether asecond computation parameter of the second group is the same as a firstcomputation parameter of the first group; in response to determiningthat the second computation parameter and the first computationparameter are the same as each other, obtaining the first result data assecond result data corresponding to the second input, without performinga neural network computation that is based on the second group; and inresponse to determining that the second computation parameter and thefirst computation parameter are different from each other, performingthe neural network computation, based on the computation parameter ofthe second group, and obtaining second result data distinguished fromthe first result data.
 12. The method of claim 11, further comprising:presenting a second frame consecutive to the first frame among theplurality of frames to the first group of the neural network model, as athird input; obtaining third result data corresponding to the thirdinput; and generating an image based on the second result data of thefirst frame and the third result data of the second frame.
 13. Themethod of claim 12, further comprising storing the third result data inthe memory.
 14. The method of claim 12, further comprising displayingthe image through a display.
 15. The method of claim 11, furthercomprising determining whether a computation structure of the firstgroup and a computation structure of the second group are the same aseach other.
 16. The method of claim 15, further comprising, when thecomputation structure of the first group and the computation structureof the second group are the same as each other, omitting the executionof a neural network computation that is based on the second group. 17.The method of claim 11, wherein the first computation parameter and thesecond computation parameter are at least one of a weight value and abias value which are used for a convolution computation.
 18. The methodof claim 11, comprising, when a computation structure of the first groupis the same as at least a portion of a computation structure of thesecond group, obtaining the first result data as intermediate resultdata of up to the same portion as the at least portion of the secondgroup.
 19. The method of claim 11, wherein a first computation layercomprised in the first group and a second computation layer comprised inthe second group comprise at least a convolution layer.
 20. The methodof claim 11, comprising providing reuse information about the firstresult data, in storing the first result data in the memory.